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FOREWORD 

The theme of qualifications frameworks and their relation to quality assurance is gaining 
urgency in the European scene as more and more countries are completing their national 
qualifications frameworks and quality assurance agencies need to take important 
decisions on how to implement them. Some of the key features of the qualifications 
frameworks are the specification of learning outcomes, processes for assessing learners' 
attainment of the expected outcomes, their relation to the ECTS, identification of transfer 
and progression routes, and recognition of prior learning. 

ENQA wanted to respond to the need of agencies to address the issue by providing 
a forum for exchange of experience and methods in implementation of qualifications 
frameworks and for a discussion on what these changes mean for the work of the 
agencies. The workshop from which this publication is an outcome of took place in 
Dublin on 9-10 February 2012. The workshop picked up and took forward work that 
ENQA had done in this area in the past years, and in particular since a workshop in 2007 
on qualifications frameworks and workshop in 2010 on learning outcomes. 

This current publication presents seven articles covering themes such as the state of 
development of qualifications frameworks, the role of agencies in the self-certification 
process, and the meaning of qualifications frameworks in external quality assurance. 

In addition, an article presenting the view of the labour market sector on qualifications 
frameworks provides depth to the range of views provided. Also recognition of prior 
learning, assessment of learning outcomes, and qualifications frameworks and learning 
outcomes in programme oriented quality assurance are addressed by this publication. 

The debate and mutual sharing within ENQA on this theme will no doubt continue in 
the coming years. In the meantime, I am confident that this publication is a useful tool 
for all agencies, as well as other interested parties, which are currently faced with new 
demands and challenges due to the implementation of qualifications frameworks and 
learning outcomes based assessment in the European Higher Education Area. 


Achim Hopbach 

President 

European Association for Quality Assurance in Higher Education (ENQA) 
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QUALIFICATIONS FRAMEWORKS 
IN THE EUROPEAN HIGHER 
EDUCATION AREA: THE STATE OF 
DEVELOPMENT AND PERSPECTIVES 
FOR THE FUTURE 

Carita Blomqvist, Head of Unit, Recognition and Comparability of Qualifications, Finnish 
National Board of Education 


INTRODUCTION 

In May 2005 the Bergen Conference of European Ministers responsible for higher 
education adopted the overarching framework for qualifications in the EHEA (QF-EHEA). 
The QF-EHEA as adopted in 2005 comprises three cycles, generic descriptors for each 
cycle based on learning outcomes, and credit ranges in the first and second cycles. 
Broadly, the QF-EHEA provides international transparency, facilitates the recognition of 
qualifications and international mobility. It has been described as a "translation device", 
which allows national qualifications to be compared. 

Ministers also committed themselves to elaborating national qualifications 
frameworks by 2010. Even if national qualifications frameworks are developed at the 
national level, one of their purposes is international comparability. The development of 
national qualifications frameworks has proved to be an important, but also demanding 
undertaking. In 2007 ministers asked the Council of Europe to take responsibility for 
coordinating the sharing of experience in the development of national qualifications 
frameworks. This request was renewed in 2009 Leuven/Louvain-la-Neuve Ministerial 
Communique. In Leuven/Louvain-la-Neuve ministers also acknowledged the importance 
of national qualifications frameworks in the implementation of lifelong learning. The 2010 
deadline proved to be too early, and the ministers indicated that they would implement 
their national qualifications frameworks and prepare for self-certification against the 
QF-EHEA by 2012 instead. 

Since 2007 the Council of Europe has chaired the Working Group on Qualifications 
Frameworks as well as the network of national correspondents. This article has 
benefitted from the draft report by the EHEA Working Group on Qualifications 
Frameworks, adopted by the Bologna Follow-up Group in January 2012. 

The start of the implementation of the qualifications frameworks was slow, and there 
was lack of progress. This was at least partly due to the fact that there was a relatively 
small pool of experience and expertise, particularly regarding the implementation of 
learning outcomes and transfer from teaching to learning orientation in higher education. 
Despite the slow start, a lot of activity and co-operation has been seen in the last few 
years at the European, national and institutional level, and in a relatively short time good 
progress has been made at all levels. By now there is a huge amount of information, 
advice and experience available for all stakeholders. 
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COMPATIBILITY AND CO-OPERATION BETWEEN OF-EHEA AND EOF 

The adoption of the European Qualifications Framework (EQF) in February 2008 has 
meant a boost also to the implementation of the QF-EHEA and the development of 
national qualifications frameworks especially in the EU member states, but also beyond. 
The two frameworks are not identical, but they are compatible and share similar 
objectives. The compatibility has been explicitly stated in the EQF Recommendationh 
According to the Recommendation, the descriptor for the higher education short cycle 
corresponds to the learning outcomes for EQF level 5, the descriptor for the first cycle 
to the learning outcomes for level 6, the descriptor for the second cycle to the learning 
outcomes for level 7, and the descriptor for the third cycle to the learning outcomes for 
level 8. This also means that it is possible to develop national qualifications frameworks 
that are compatible with both frameworks. 

For implementation, close co-operation has been established between the European 
Commission and the Council of Europe, as well as in member states. Co-operation in 
member states is supported by the co-operation between the European networks: the 
network of national correspondents and the National Coordination Points of the EQF. 

National qualifications frameworks developed within the European higher education 
area should be compatible with the QF-EHEA. This compatibility is demonstrated and 
verified through the agreed self-certification process. This means that the competent 
national authorities (with the support of international experts) carry out a verification 
process and issue a document certifying the compatibility of their national framework 
with the QF-EHEA. A similar process, called the referencing process, has been agreed 
upon to confirm the reference of member states' national qualifications levels to the EQF. 
For this process, the EQF Advisory Group has agreed on a set of criteria and procedures 
to guide the process. The starting point for the EQF Advisory Group in this work was 
the self-certification process which had been established for the QF-EHEA earlier. Qne 
of the important success factors for the compatibility of the QF-EHEA and the EQF are 
self-certification and referencing processes. They share not identical, but rather similar 
criteria and procedures and it is possible for countries to combine both processes in one 
report. 

LINKING QUALIFICATIONS FRAMEWORKS, QUALITY ASSURANCE AND 
RECOGNITION 

Qualifications frameworks link closely to quality assurance and recognition. Qne of 
the common issues for all of them is learning outcomes, even if the perspectives are 
different. 

Using learning outcomes is a basic requirement for the design and implementation 
of qualifications frameworks. The learning outcomes expressed in the national 
qualifications frameworks are essential for aligning the national qualifications framework 
with the QF-EHEA and the EQF. 

Qn the other hand, even if only the first steps have been taken, recognition 
authorities and practice focus more and more not only at the input factors, like time 
and number of credits, but also on the learning outcomes. Self-certified/referenced 
national qualifications frameworks are considered a tool to facilitate recognition of 
foreign qualifications. At least they should help recognition authorities avoid asking 
detailed questions about quality, level, and workload, and they would considerably 
help recognition authorities understand the learning outcomes provided by foreign 
qualifications. Qne of the approaches for defining substantial difference relates to the 

1 Recommendation of the European Parliament and of the Council on the establishment of the European Qualifications 

Framework for lifelong learning (April 2008) 
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learning outcomes of the qualification. The legal basis for recognition is the Council of 
Europe/UNESCO Convention on the recognition of qualifications concerning higher 
education qualifications (Lisbon Recognition Convention). The Bureau of the Committee 
of the Lisbon Recognition Convention has started work on subsidiary text to the 
Convention - a recommendation on the role of qualifications frameworks in facilitating 
recognition. The aim is to have the subsidiary text adopted in the next intergovernmental 
committee meeting in 2013. 

Again, quality assurance experts are important stakeholders in the design and 
implementation of the qualifications frameworks. Transparent national quality assurance 
is a prerequisite for the implementation and functioning of the national and European 
qualifications frameworks. International self-certification and referencing processes 
can be considered as part of the overall quality assurance of qualifications frameworks 
at national and European levels. As one of the purposes of the national qualifications 
frameworks is to improve the quality of teaching and learning, quality assurance is seen 
as increasingly linked also to learning outcomes and discussion about the role of quality 
assurance in different stages of the learning process as well as in assessing learning 
outcomes is emerging. 

CHALLENGES AND OPPORTUNITIES IN IMPLEMENTATION 

Even if developing and designing a national qualifications framework is a demanding 
task, which requires a lot of discussions, communication, consistency, and stakeholder 
participation, it is even more challenging to actually implement the national qualifications 
framework. It seems that proper, inclusive, and consistent work in designing the 
framework also makes the implementation easier. A wide basis for stakeholder 
involvement is important. 

It is important that the purposes and aims of the national qualifications framework 
have been thoroughly considered. In general, the aims and purposes of the national 
qualifications frameworks are very similar but there can be national priorities. It is up 
to national stakeholders to consider whether our national qualifications framework is 
mainly seen as a transparency tool: aid for international comparability and recognition, 
and recognition of non-formal and informal learning - or whether the emphasis is placed 
on the shift towards learning outcomes in higher education. Other common purposes are 
the enhancement of access and lifelong learning. Overall improvement of the quality of 
teaching and learning should always be one of the purposes of the national qualifications 
frameworks, even if it were not always clearly articulated. According to self-certification 
and referencing reports it seems to be common that national qualifications frameworks 
serve several purposes. This, again, means that they are useful instruments for many 
purposes. 

Qualifications frameworks are already making a difference in promoting learning 
outcomes. It is important to see that this does not happen in isolation, but the link to 
quality assurance and recognition is maintained. The connection between qualifications, 
learning outcomes, qualifications frameworks and quality assurance is such that they 
should not be regarded as discrete aspects to be worked on in isolation. Qualifications 
frameworks are also tools which can be used to facilitate validation in general and 
recognition of non-formal and informal learning specifically. 

At the European level it is important to pay attention to consistency, when it comes 
to referencing and self-certification of qualifications. Lately, access qualifications have 
been discussed and the discussion concerning short cycle qualifications is emerging. 
While it is up to the national authorities to decide on the levels of qualifications, it is 
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internationally difficult to understand decisions which have been made against broadly 
accepted trends, especially when it comes to qualifications concerning access to higher 
education and their established recognition practice. In the long run, inconsistency in 
referencing or self-certifying certain qualifications that are typical for all countries could 
even undermine the overall importance of national and the European frameworks. 

WAY AHEAD 

It cannot be emphasised too much that we still need better understanding of the 
dynamics and links between quality assurance, qualifications frameworks, learning 
outcomes, and recognition. They can all be seen as tools that support and complement 
each other and should be developed and implemented jointly. 

Experience in the design and implementation of qualifications frameworks is gained 
all the time at institutional, national, and European levels. National and European 
qualifications frameworks must not become static - they need to evolve and develop. 
Also level descriptors and learning outcomes develop and change over time. Should 
this have an influence also at the European level and the European frameworks? At 
the European level we should look at the synergies between different Bologna and 
transparency tools. This may also lead to a renewal of self-certification and referencing 
processes, which, again, have to be looked at critically at some stage. Do they serve their 
purpose and are they up-to-date? 

We should also try to make sure that European, but especially national qualifications 
frameworks, become more visible and bring benefits to end users, be they individuals, 
higher education institutions, academic staff, or employers. So far a lot of attention 
has been paid to structures and institutions, but in the end the acceptance of and trust 
towards national and European qualifications frameworks by end users will be decisive 
for the overall success of qualifications frameworks. 
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THE ROLE OF QUALITY 
ASSURANCE AGENCIES IN 
THE SELF-CERTIFICATION OF 
COMPATIBILITY OF NOES WITH 
THEQF-EHEA 

Bryan Maguire, Head of Qualifications Services Designate, HETAC, Ireland 


INTRODUCTION 

Two of the most potent and concrete policy tools resulting from the Bologna Process 
reforms are quality assurance, as epitomised by the European Standards and Guidelines 
for Quality Assurance in Higher Education (ESG)^, and the widespread use of learning 
outcomes in higher education, of which the highest level expression is the Qualifications 
Framework of the European Higher Education Area (QF-EHEA). This paper will examine 
a key process in which these two policy tools interact, namely the verification of 
compatibility of national qualifications frameworks (NQFs) with the QF-EHEA by means 
national self-certification. 

It is axiomatic that quality assurance is first and foremost a prerogative of the higher 
education institutions (HEIs) but independent, external quality assurance agencies 
(QAAs) are also indispensable actors in the European model and it is these agencies 
that constitute the membership of ENQA. The growth of quality assurance in higher 
education predated the Bologna Declaration but as the reforms flowing from that took 
shape, it was clear that quality assurance was to play an important role. The political 
commitment to quality assurance took concrete form when the ministers responsible 
for higher education endorsed the ESG at their meeting in Bergen in 2005. The ESG 
confirmed the role of QAAs as having responsibility for the external dimension of 
quality assurance (Part 2 of ESG) and identified key characteristics and accountability 
mechanisms for the QAAs themselves (Part 3 of ESG). 

An interest in specifying learning outcomes for higher education and the use of 
qualifications frameworks as an organising device to support this task also predated the 
Bologna Declaration. This work took on renewed urgency as the reforms into two and 
later three cycles were rolled out across the continent. This led a number of countries 
to organise a project called the Joint Quality Initiative where they worked out a set of 
descriptors for the qualifications awarded at the end of the cycles. From there the next 
step was to consider bringing these Dublin Descriptors, as they came to be called, into an 
overarching meta-framework of qualifications for the European Higher Education Area 
(QF-EHEA). A Working Group (WG-QF-EHEA), chaired by Sjur Bergan of the Council of 
Europe, prepared a detailed proposal which was also adopted at the Bergen ministerial 
meeting in 2005. QA agencies contributed strongly to this working group both in the 
initial development of the QF-EHEA and in its subsequent implementation. 

As well as adopting the Dublin descriptors, the ministers at Bergen also endorsed 
criteria (List 1) and procedures (List 2) for verifying the compatibility of national 

2 http://www.enqa.eu/pubs_esg.lasso 
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qualifications frameworks with overarching QF-EHEA and set out an ambitious timeline 
that all member countries should have completed this verification by a process of self- 
certification by 2010. No European level authority was established to pass judgment on 
these verifications, though the Bologna Follow-up Group did request WG-QF-EHEA to 
continue to monitor and support the implementation of the QF-EHEA. 

Since both the ESG and the QF-EHEA proposals were developed in advance of the 
Bergen Summit and neither set of proposers could be sure that the other would actually 
be adopted, the two documents do not explicitly cross-reference each other. However, 
there are important substantial linkages. Guideline 1.2 of the ESG states that "the quality 
assurance of programmes and awards are expected to include [...] development and 
publication of explicit intended learning outcomes", thereby invoking learning outcomes, 
the key concept underpinning the QF-EHEA. Similarly criterion 5 for self-certification 
anticipates convergences in the development of national quality assurance systems that 
are expressed in the ESG. 

LIST 1 - CRITERIA FOR VERIFYING COMPATIBILITY OF NOFS WITH OF-EHEA 

1. The national framework for higher education qualifications and the body or bodies 
responsible for its development are designated by the national ministry with 
responsibility for higher education 

2. There is a clear and demonstrable link between the qualifications in the national 
framework and the cycle qualification descriptors of the European framework 

3. The national framework and its qualifications are demonstrably based on learning 
outcomes and the qualifications are linked to ECTS or ECTS compatible credits 

4. The procedures for inclusion of qualifications in the national framework are 
transparent 

5. The national quality assurance system for higher education refers to the NQF 
and is consistent with the Berlin Communique and any subsequent communique 
agreed by ministers in the Bologna Process 

6. The national framework, and any alignment with the European framework, is 
referenced in all Diploma Supplements 

7. The responsibilities of the domestic parties to the national framework are clearly 
determined and published 

LIST 2 - PROCEDURES FOR VERIFYING COMPATIBILITY OF NOFS WITH 
OF-EHEA 

1. The competent national body/bodies shall certify the compatibility of the national 
framework with the European framework. 

2. The self-certification process shall include the stated agreement of the quality 
assurance bodies in the country in question recognised through the Bologna 
Process 

3. The self-certification process shall involve international experts 

4. The self-certification and the evidence supporting it shall be published and shall 
address separately each of the criteria set out 

5. The ENIC and NARIC networks shall maintain a public listing of states that have 
confirmed that they have completed the self-certification process 

6. The completion of the self-certification process shall be noted on Diploma 
Supplements issued subsequently by showing the link between the national 
framework and the European framework 
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TYPICAL STEPS IN THE DEVELOPMENT OF A NATIONAL 
OUALIFICATIONS FRAMEWORK 

It was recognised that the development of a NQF is a national prerogative and that 
different national systems and traditions would take slightly different paths towards 
its achievement. By way of guidance the WG-QF-EHEA outlined a set of typical steps 
required in the development of national qualifications frameworks, which it included 
in its progress report to the ministerial conference in London in 2007. That report also 
included other guidance based on lessons learned from the pilot self-certifications 
carried out in 2006 in Ireland and Scotland: 

1. Decision to start 

2. Setting the agenda: the purpose of our NQF 

3. Organising the process 

4. Design 

5. Consultation 

6. Approval 

7. Administrative set-up 

8. Implementation at institutional/programme level 

9. Inclusion of qualifications in the NQF 

10. Self-certification of compatibility with the EHEA framework 

COUNTRIES THAT HAVE COMPLETED SELF-CERTIFICATION 

The initial target of the ministers, adopted in Bergen in 2005, was for all countries to 
have completed verification by means of self-certification by 2010, in time for the formal 
inauguration of the EHEA. This timetable was always ambitious and in the Leuven 
Communique (2009), on the advice of the WG-QF-EHEA, the ministers extended 
the deadline. To date, ten NQFs have completed self-certification and given the pace 
of national developments it is expected that many more, though certainly not all 47 
signatory states, will do so in 2012. A significant factor that has complicated, and in 
some cases inevitably delayed, completion has been the introduction of the European 
Qualification Framework for Lifelong Learning (EQF-LLL) in 2008. In some countries the 
verification of the QF-EHEA is carried out in a combined exercise with the referencing of 
the NQF to the EQF-LLL, which necessarily involves a wider range of stakeholders. 

COUNTRIES THAT HAVE COMPLETED SELF-CERTIFICATION AS OF FEBRUARY 
2012 

• Belgium - Flemish Community, February 2009 

• Denmark, November 2009 

• Germany, January 2009 

• Ireland, November 2006 

• Malta, August 2009 

• The Netherlands, February 2009 

• Portugal, May 2011 

• Romania, November 2011 

• UK - Scotland, December 2006 

• UK - England, Wales and Northern Ireland, February 2009 
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PARTICIPATION OF OUALITY ASSURANCE AGENCIES IN SELF- 
CERTIFICATION OF NOFS 

The participation of quality assurance agencies in the implementation of NQFs, including 
verification, was anticipated from the outset. The second required procedure in self- 
certification states that "the self-certification process shall include the stated agreement 
of the quality assurance bodies in the country in question recognised through the 
Bologna Process". Nevertheless, the different roles, capacities, and priorities of various 
QA agencies within national contexts, have led them to play different parts in self- 
certification. Regardless of the how central or otherwise the QA agency is to the conduct 
of the self-certification exercise itself, each agency can be expected to form a view about 
the individual criteria from the perspective of quality assurance. 

AGENCIES' PARTICIPATION IN SELF-CERTIFICATION OF NOFS 

Coordinating role in verification: 

• NVAO(BE-FL) 

• NVAO(NL) 

• EVA(DK) 

• QAA(UK) 

• ARACIS(RO) 

• NQAI(IE) 

Committee/steering group member: 

• German Accreditation Council (DE) 

• lUQB, HETAC (IE) 

• ACE Denmark 

Other role: 

• VLIR, VLHORA(BE-FL) 

• MQC*(MT) 

• A3ES*(PT) 

*Not an ENQA member. 

Criterion 1 states that "the national framework for HE qualifications and the body or 
bodies responsible for its development are designated by the national ministry with 
responsibility for HE". The rather initial straightforward question here is whether the 
QA agency is satisfied that its role in relation to self-certification has been properly and 
formally defined. If it is de facto responsible, has this been formally determined by the 
minister? 

Criterion 2 - "there is a clear and demonstrable link between the qualifications in the 
national framework and the cycle qualification descriptors of the European framework" 

- is the core issue for verification. The question that a QA agency must ask is whether 
the outcome of the technical analysis supporting a positive response to this criterion 
corresponds to the operational understanding on which its quality assurance activities 
are based. 

Criterion 3 - "the national framework and its qualifications are demonstrably based 
on learning outcomes and the qualifications are linked to ECTS or ECTS compatible 
credits" - is an example of an issue that depends to a considerable extent on the role of 
the QA agency. Is it the QA agency that has responsibility for demonstrating the basis on 
learning outcomes or does this rest more with the HEIs? Is the QA agency responsible for 
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articulating learning outcomes for disciplines at a national level in the form of standards 
or guidelines? 

Criterion 4 - "the procedures for inclusion of qualifications in the national framework 
are transparent" - goes to the heart of quality assurance. The two basic models are 
either that the agency has responsibility for contributing to this transparency through 
programme accreditation, or that its quality assurance of institutions ensures that the 
higher education institutions are transparent. 

Criterion 5 relates even more explicitly to quality assurance and hence to the QA 
agencies: "the national quality assurance system for higher education refers to the 
NQF and is consistent with the Berlin Communique and any subsequent communique 
agreed by ministers in the Bologna Process". At this stage in development this means 
that a country must have implemented the ESG and have had its QA agency or agencies 
reviewed under Part 3 of the ESG. 

Criterion 6 - "the national framework, and any alignment with the European framework, 
is referenced in all Diploma Supplements" - may or may not be a concern of the 
QA agency. In some systems the QA agency has a role in guiding or auditing the 
implementation of the diploma supplement. 

Criterion 7 - "the responsibilities of the domestic parties to the national framework 
are clearly determined and published" - may be more difficult to judge than appears at 
first sight: this is particularly the case in countries where roles and responsibilities are 
changing or are highly contested. The QA agency has to ensure that its role is accurately 
expressed in the documentation used to support the answer to this criterion. 

CONCLUSIONS 

In conclusion, verification by self-certification is a process in which each QA agency has 
a significant and obligatory role to play. However, self-certification is only a staging point 
along the way towards the dream of the EHEA. If we are to realise the objectives of the 
QF-EHEA in relation to transparency, mobility and recognition further work is required. 
As the "critical mass" of countries with frameworks grows the expectations of partner 
countries and other stakeholders is that NQFs should be implemented with quality 
assurance being used to demonstrate that programmes are based on intended learning 
outcomes and that qualifications are awarded on the basis that these outcomes have 
been achieved. 

REFERENCE WEBSITES 
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http://www.ehea. inf o/article-details.aspx?Articleld=69 

Qfficial site of the European Qualifications Framework for Lifelong Learning 
http://ec.europa.eu/eqf/ 


13 


ROLE AND MEANING OF 
OUALIFICATIONS FRAMEWORKS IN 
EXTERNAL OUALITY ASSURANCE 

Karin Jdrplid Linde, Assistant Head of Department, Swedish National Agency for Higher 
Education 


A NEW MODEL FOR QUALITY ASSURANCE OF HIGHER EDUCATION 

The Swedish National Agency for Higher Education (the Agency), established in 
1995, has the national responsibility for quality assurance of higher education through 
quality evaluation of first, second and third cycle study programmes. The Agency is 
also responsible for the appraisal of higher education institutions entitlement to award 
qualifications. This article will focus on the system for quality evaluations of first and 
second cycle programmes. A new model, focusing on student attainment of the learning 
outcomes specified in the Higher Education Ordinance was introduced in 2011. No 
evaluation reports have yet been published, and the question is: can results of an 
academic study programme be measured? If yes: how can that be done? And last, but 
not least: is this European quality assurance of the 21st century? 

But first, some background information. 

QUALIFICATIONS FRAMEWORKS IN SWEDEN 

Higher education in Sweden is to a large extent regulated by the Higher Education Act 
and the Higher Education ordinance. In 2007 major changes were made in order to 
align the Swedish qualification system with the Bologna Process and the Qualifications 
Framework of the European Higher Education Area (QF-EHEA) adopted by the ministers 
in Bergen in 2005. A National Qualifications Framework was established, and as from 1 
July 2007 all qualifications are defined in terms of learning outcomes and workload. Each 
qualification is ascribed to one of three cycles: first, second and third. The requirements 
for each cycle are specified in the Higher Education Act and correspond to the QF-EHEA. 
In line with the Bologna Process and the goal of increased employability for students, 
qualification descriptors were introduced with specified outcomes of what "a learner is 
expected to know, understand and/or be able to do at the end of a period of learning''^ 
These qualification descriptors'* have become a very important tool, or you might say, the 
tool, in quality assurance of higher education in Sweden.** 

QUALITY ASSURANCE IN SWEDEN - AN IDEOLOGICAL APPROACH 

In March 2010 the Government presented the bill Fokus pa kunskap - kvalitet i den hogre 
utbildningen [Focus on knowledge - quality In higher education] which was enacted by 
the Riksdag on 3 June 2010.^ According to the government, greater autonomy should 
be given to the higher education institutions.^ Qne important consequence was that the 

3 A Framework for Qualifications of the European Higher Education Area, Ministry of Science, Technology and Innovation, 
Denmark, 2005, p. 166 

4 The qualification descriptors can be found in Annex 2 to the Higher Education Ordinance (1993:100), the Annex to the 
Ordinance for the Swedish University of Agricultural Sciences (1993:221) and the Annex to the Ordinance for the 
Swedish National Defence College (2007:1164). 

5 Govt. Bill Ny varld - ny hogskola [New world - New university] (2004/05:162) p. 99. 

6 Govt. Bill Focus on knowledge - quality in higher education. 2009/10:139, bet. 2009/10:UbU20, rskr. 2009/10:320. 

7 Govt. Bill En akademi i tiden [Academia for this day and age- greater freedom for universities and other higher education 
institutions] (Govt. Bill 2009/10:149). 
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institutions should take full responsibility for the development and quality assurance 
of their activities. The national quality assurance system, for which the Agency is 
responsible, was no longer to focus on the internal processes of the higher education 
institutions, but on the results of courses and study programmes. The National Agency 
was to assess to what extent the students' achieved learning outcomes correspond to 
the intended learning outcomes laid down in the qualification descriptors specified in the 
Higher Education Ordinance. 

In previous systems for quality assurance in Sweden, excellence has not been 
rewarded. No differentiation has been made between higher education institutions 
whose programmes barely achieve expected standards and those whose programmes 
produce excellent results. In the Government's view, this is not reasonable. Therefore, 
higher education institutions with high quality programmes will be rewarded through 
increased funding. The evaluations will provide a basis for the Government's allocation of 
a small percentage of that extra funding. 

THE EVALUATION MODEL 2011-2014 - HOW DOES IT WORK? 

In the peer review based quality evaluation system that was introduced in 2011, the 
Agency assesses the outcomes of study programmes. This is done by appraising the 
degree to which the students achieve the outcomes laid down in the qualification 
descriptors. The National Agency assesses the extent to which the students' achieved 
learning outcomes correspond to the intended learning outcomes. For each evaluation 
the peer review team makes a selection of the outcomes listed in the Higher Education 
Ordinance on which to base the subsequent assessment of the material. As the 
outcomes vary in numbers and complexity, the Agency has set up guidelines for the 
selection procedure in order to assure equivalence in the evaluations. The peer review 
team suggests a selection of outcomes which are discussed with the higher education 
institutions before a final decision is made by the Agency. 

WHAT KIND OF INFORMATION IS USED? 

Four different assessment factors are taken into account: the students' independent 
projects (degree projects), the higher education institution's own self-evaluations, 
questionnaires sent to previous students, and the students' impression of the outcomes 
of their programmes of study in relationship to the outcomes laid down in the 
qualification descriptors. The students' independent projects together with the learning 
outcomes accounted for in the self-evaluations provide the main basis for the overall 
assessments. 

STUDENTS' INDEPENDENT PROJECTS 

When the Bologna reforms of higher education in Sweden were implemented in 2007, 
the requirement of an independent project (degree project) for the award of virtually 
all qualifications was introduced. The result of the independent project is seen as one 
way of confirming that students have achieved the intended outcomes. Therefore, 
in the national quality assurance system, assessment of a selection of anonymous 
independent projects is important to ascertain student attainment in relation to intended 
outcomes as indicated in the qualification descriptors. According to a statistical model 
a random selection of a maximum of 24 and a minimum of 5 projects from each study 
programme is assessed. This appraisal is not a review of individual students but a means 
of assessing the results of a study programme on the basis of the outcomes laid down 
in the qualification descriptors. It is the aggregate quality of a programme's independent 
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projects that is to provide the basis of the evaluation and not specific excellent or poor 
productions. Thus, it is an important part of the process, as it is a clear indication of the 
extent to which students have attained the expected academic outcomes. However, the 
importance of these projects varies between different programmes, and it is sometimes 
relatively minor when it comes to some of the professional degrees, which must be taken 
into account in the evaluations. In these cases the other assessment materials, especially 
the evidence of outcome attainment in the self-evaluations, become more important. 

THE HIGHER EDUCATION INSTITUTIONS' SELF-EVALUATIONS 

The self-evaluations submitted by the higher education institution serve two purposes. 
The first is to enable a broader and more comprehensive presentation of outcomes 
than a selection of independent projects can offer. In their self-evaluations, therefore, 
the institutions should present, analyse, and assess the outcomes attained in relation 
to all the outcomes to be taken into account in the evaluation. They should include 
specific evidence that outcomes are attained rather than references to plans, conditions 
and processes. Evidence may be presented in terms of summaries and analyses (with 
illustrated examples) of students' essays or laboratory reports. The institutions' own 
analysis of the independent projects may also be used. Some reference to prerequisites 
and processes may, however, be presented to demonstrate how the institution 
guarantees student attainment of these outcomes. Secondly, circumstances that have 
manifest significance for the results of a programme, e.g. the qualifications of teachers 
and their availability, as well as students' preconditions, can be taken into account. In 
cases where few independent projects are available or where they play a minor role in the 
programme, the self-evaluation becomes proportionately more important. 

STUDENTS' EXPERIENCES 

Students' experiences are collected in interviews with students approaching the end of 
their studies. The aim is to find additional evidence regarding the extent to which the 
programme contributes to their attainment of its intended outcomes as indicated in the 
qualification descriptors. 

ALUMNI QUESTIONNAIRES 

Questionnaires for alumni are intended to provide information about whether a 
programme attains the outcome of usefulness in the labour market. They may also 
provide indications about whether those now employed consider that they attained the 
intended outcomes. The questionnaires are sent primarily to those who graduated two 
years prior to the evaluation, and who can thus be assumed to have gained labour market 
experience, while at the same time their studies are not too distant. Questionnaires are 
sent to no less than 20 alumni. A response rate of at least 50 per cent is required for 
results to be considered reliable. 

A THREE-LEVEL SCALE FOR THE OVERALL ASSESSMENT 

The overall assessment of an evaluation is presented on a three-level scale: 

1 . Very high quality: the study programme displays a very high degree of achievement of 
outcomes 

2. High quality: the study programme displays a high degree of achievement of outcomes 

3. Inadequate quality: the study programme does not achieve all outcomes 
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The panel submits its overall assessment of each study programme to the Agency using 
the three-level scale. This evaluation must clearly indicate how well the students are 
considered to have achieved the outcomes evaluated. The grounds for each evaluation 
must be provided by the panel. On the basis of the assessment panel's report, the 
Agency reaches a decision on the overall assessment to be awarded to each programme 
of study. 

Those that are given the overall assessment of Very high quality or High quality are 
considered to have attained the quality standards for higher education. Those that fail to 
attain the required standards are given the evaluation Inadequate quality, in which case 
the Agency extends the higher education institution's entitlement to award the relevant 
qualification conditionally. These programmes will then be reviewed within the following 
year before the Agency decides whether or not the entitlement should be revoked. A 
small percentage of the higher education budget provided by the Government will be 
reserved for institutions with programmes that display Very high quality. This funding will 
be made available as from 2013, after the first results of the four-year cycle have been 
presented. 

SOME POSITIVE SIDE EFFECTS 

The evaluation system will contribute to increased knowledge and awareness of the 
national qualification descriptors. What we have seen so far is that higher education 
institutions review how intended learning outcomes for individual courses are linked to 
the qualification descriptors. 

Higher education institutions have been seen to work harder to improve their internal 
quality assurance. For example, they do their own pre-evaluations, in particular when 
it comes to the assessment of independent projects. In some cases institutions have 
decided to close down a programme facing evaluation. 

CHALLENGES 

A minimum of five independent projects (or the programme to be evaluated 

In an output oriented evaluation system where student's results are an important part of 
the assessment, the independent project has proved to be a key element in the evaluation 
process. A random selection of between 5 and 24 independent projects is appraised for 
each programme evaluated. No programmes with less than five independent projects will 
be part of this evaluation system. How they will be evaluated, remains to be decided. 

An output oriented self-evaluation 

The most important part of the self-evaluation is where the higher education institution 
has the opportunity to present, analyse and assess the outcomes achieved in relation 
to the targets in the qualification descriptors. A minor part of the self-evaluation treats 
prerequisites, such as teacher qualifications. This shift, towards an output oriented 
self-evaluation has caused difficulties for the higher education institutions as to how 
to present evidence of attainment of outcomes. Student's intended learning outcomes 
are usually unproblematic to describe, but how is evidence of the actual achievement 
presented in a self-evaluation? This has been one of the key issues for the higher 
education institutions in the self-evaluation process. 

Are we assessing students more than programmes? 

This is probably one of the most frequent criticisms of the evaluation system. To us, it is 
obvious that if the qualification descriptors, established in line with the QF-EHEA, are to 
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be at the centre of the evaluation processes, it is necessary to include students' actual 
goal attainment in the evaluation process. And, as has already been stated, the appraisal 
of the students' independent projects is not about redoing the work of the examiner, but 
to look for goal attainment in relation to the outcomes in the qualification descriptors. It 
is the aggregate quality of the programme's independent projects that is to provide the 
basis of the evaluation and not specific excellent or poor productions. The panel assesses 
the degree of outcome achievement in the independent projects, in the self-evaluation 
or in any of the other assessment material. This evaluation system is not about assessing 
students, but putting students' knowledge and employability at the heart of quality 
assurance. 

CONCLUDING REMARKS 

Now, back to the initial questions: yes, we believe this can be done. We are at the 
beginning of the process and continuous review will help us learn and develop best 
practice. We evaluate learning outcomes in line with the Bologna Process for the sake of 
the students and increased employability. 

REFERENCES 

A Framework for Qualifications of the European Higher Education Area (page 166). Ministry 
of Science, Technology and Innovation, Denmark, 2005. 

Academia for this day and age- greater freedom for universities and other higher education 
institutions. Government Bill 2009/10:149. 

Focus on knowledge - quality in higher education. Government Bill 2009/10:139, bet. 
2009/10:UbU20, rskr. 2009/10:320. 

General Guidelines for Self-Evaluation in the Swedish National Agency for Higher Education's 
Quality Evaluation System, 2011-2014. 2011:11R. Swedish National Agency for Higher 
Education 2011. 

New World - New University (page 99). Government Bill 2004/05:162. 

Swedish Code of Statutes (SFS) No. 1992:1434, Higher Education Act. Ministry of Education 
and Research, Sweden (www.hsv.se/lawsandregulations). 

The Swedish National Agency for Higher Education's quality evaluation system 2011-2014. 
2011:3R. Swedish National Agency for Higher Education 2011. 


18 


QUALITY ASSURANCE AND 
QUALIFICATIONS FRAMEWORKS: A 
STAKEHOLDER VIEW 

Tony Donohoe, Head of Education Policy, Irish Business and Employers Confederation 


INTRODUCTION 

Access to a skilled workforce is a pressing challenge that companies face to maintain 
and improve their competitive position at global level. Around the world, we witness an 
unprecedented competition between companies to attract talent. It is critical that Europe 
has a work force that is mobile and equipped with a set of skills and competencies that 
are responsive to actual labour market demands. Therefore we need transparent and 
comparable qualifications, irrespective of the way the competencies have been acquired. 

This short paper offers an employer's view, of quality assurance and qualifications 
frameworks. In order to provide a context, it starts with a brief recent history of the 
evolution of qualifications and quality assurance in Ireland. It explores some of the 
benefits for policy makers, employers, employees and funders. The paper also suggests 
that there are significant challenges at a national and European level before the potential 
of qualifications frameworks to underpin a lifelong learning society can be fully realised. 

EVOLUTION OF OUALIFICATIONS AND OUALITY ASSURANCE IN 
IRELAND 

Towards the end of the 1990s, there was a growing realisation of the need for a 
coherent national policy approach to qualifications in Ireland. It was based on an 
acknowledgement that the present and future workforce would need to constantly 
update its knowledge, skills and competence. This would require a more flexible 
system of qualifications and greater clarity in the meaning of qualifications. As the 
global workforce was becoming more mobile, there was also a need for international 
comparison and alignment of qualifications. Meanwhile developments, such as the 
Lisbon Strategy, the Copenhagen process, the Bologna framework and the European 
Qualifications Framework (EQF) provided further impetus to the development of the a 
National Framework of Qualifications (NFQ). Key milestones in the process were: 

• Qualifications (Education and Training) Act, 1999 

• The establishment of three new organisations to bring coherence to the system. 
These were (a) the Further Education and Training Awards Council (FETAC) which 
would have responsibility for making and promoting awards at levels 1 - 6® on the 
NFQ; (b) the Higher Education and Training Award Council (HETAC) which would 
have responsibility for levels 6-10; (c) the National Qualifications Authority of 
Ireland (NQAI) which would have overall responsibility for the establishment and 
maintenance of a framework of qualifications. 

• The introduction of the National Framework of Qualifications (NFQ) in 2003. The 
NFQ is built on an architecture of 'levels', 'award-types', and 'named awards'. There 
are ten levels and four classes of award-type (major, minor, supplemental and 
special-purpose). The framework includes 16 major award-types defined for the 


8 The Irish NFQ predated the EQF and was based on a 10-level framework. These levels have since been referenced to the 
EQF. 
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framework (e.g. advanced certificate, honours bachelor degree). A 'named award' is 
what a learner actually receives (e.g. Advanced Certificate in Craft - Electrical.) 

• Ireland verified the compatibility of its National Framework of Qualifications with 
the Bologna Framework in 2006 and completed the referencing of its levels against 
those of the EQF in June 2009. 

In 2008, the Irish Government decided to amalgamate the NQAI, FETAC and HETAC 
into a single agency. Qualifications and Quality Assurance Ireland. The new organisation 
will also take responsibility for the external quality assurance review of the universities, 
a function which is currently performed by the Irish Universities Quality Board and the 
Higher Education Authority. 

BENEFITS FOR EMPLOYERS 

The development of NFQ has helped to deepen Irish employers' understanding of the 
education and training system so that they can attract and develop staff that possess the 
skills needed for business success. As in many European countries, the education and 
training systems in Ireland had developed along parallel lines. Employers do not make 
this distinction. The framework has been a valuable tool for Irish companies in comparing 
qualifications, whatever their origin. It has also allowed employers to: 

• Specify the level of knowledge, skill and competence required for occupational 
roles. 

• Evaluate qualifications presented. In this regard, the NFQ widened the pool of 
suitable applicants as individuals with comparable qualifications to those stipulated 
and those with comparable foreign qualifications could be considered for positions. 
The NQAI offered a service called Qualifications Recognition, which provided 
employers and recruiters with a means of comparing and contrasting foreign 
qualifications with qualifications in the Irish system. 

• Simplify their applications procedure. 

• Connect training and development with business needs - the ten levels of the NFQ 
are based on learning outcomes; what an individual knows, understands and is able 
to do following successful completion of a programme of study. Learning outcomes 
and the clarity provided by the NFQ regarding access, transfer and progression 
opportunities has helped employers to plan and assess staff training and education 
needs, by pitching and fitting the right education and training with both individual 
and business needs, thereby increasing return on their investment. 

• Get their in-company education and training certified. All qualifications on the NFQ 
are quality assured and the Framework supports the development and certification 
of all learning wherever it takes place; this includes the workplace. In-company 
training can lead to nationally and internationally recognised qualifications, thus 
rewarding employees for their participation in continuous professional development 
programmes. 

• Secure funding for workplace training. Increasingly, public funding for workplace 
training and development is aimed at supporting up-skilling initiatives that lead to 
qualifications recognised through the NFQ. 

• Support performance management by using qualifications on the NFQ to motivate 
their workforce to maintain high performing work environments. 

• Become an employer of choice. Using the NFQ as a workplace resource 
demonstrates the company's commitment to investing in their most important 
asset and helps them to attract, retain and motivate the talent needed for success. 
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While this paper primarily offers the employer perspective, it is also noteworthy 
that the NFQ enables learners to achieve qualifications in different ways, in formal, 
non-formal and informal learning contexts. It also emphasises credit accumulation and 
transfer, and the achievement of smaller packages of learning. The framework removes 
the distinction between qualifications attained for learning on the job and off the job. It 
also provides access to learning opportunities and improves information provision. 

Overall the NFQ has provided a coherent framework on which to base Ireland's 
national skills strategy. This presents a vision of Ireland in 2020 with a well-educated 
and highly skilled population which contributes to a competitive, innovation-driven, 
knowledge-based, participative and inclusive economy. The strategy states that by 2020: 

• 48% of labour force should have qualifications at NFQ Levels 6-10 (3rd/4th level) 

• 45% should have qualifications at Levels 4 & 5 (upper secondary); 

• The remaining 7% will have qualifications at Levels 1-3 (lower secondary or below) 

• Ireland should aim to build capability at fourth level and double its PhD output 
(Level 10) by 2013 

The framework enables the government agency to tangibly measure its progress in 
achieving these ambitions. 

CHALLENGES 

This account does not seek to minimise the significant challenge facing qualifications and 
quality assurance agencies, particularly at a European level. The most significant of these 
challenges is the need to develop awareness of the frameworks amongst all stakeholders. 
This task is sometimes made more difficult by the obscure language that surrounds 
the subject. Qualifications and quality assurance is a highly technical and specialised 
business with its own conceptual framework. However, it is in danger of becoming an 
industry in itself, which is divorced from the realities of the workplace and leaner needs. 

I would suggest that this challenge has been acknowledged in Ireland and a lot of effort 
has been invested in translating the technicalities of the subject into something that 
employers and their staff can understand. A recent survey by the Irish Business and 
Employers Confederation indicated that 70% of Irish employers considered accreditation 
on the NFQ as a significant factor when selecting an external training programme. 

Given its relatively short (nine year) history, this is a remarkable achievement. 

However there are challenges. For example, in some sections of the third level sector, 
there is a perceived conflict of interest between labour market orientation and higher 
education values. There is a belief amongst some academics that qualifications 
frameworks encourage a limited 'reductionist' view of education. Again, in Ireland, there 
has been significant progress in addressing these concerns. The University Framework 
Implementation Network was jointly established by the National Qualifications Authority 
of Ireland and the Irish Universities Association in 2007 (see http://www.nfqnetwork.ie). 
The purpose of the network is to deepen the implementation of the NFQ within the 
university sector, primarily through the exchange of experience and practice between 
members. The network has developed common principles and approaches to framework 
implementation, and produced guidelines on the design of programmes and the use and 
assessment of learning outcomes. 

There are also missed opportunities with regard to employers fully using the 
framework, particularly to accredit their own programmes. Irish and European companies 
spend significant recourses on in-company training. Yet the IBEC survey revealed 
that only 22.5% of companies were seeking accreditation for their internal training 
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programmes. This is not heiped by the perception that the accreditation process is 
compiex and time-consuming. There wiii aiways be a heaithy tension between awarding 
bodies, who are concerned with quaiity, and training providers who are seeking 
accreditation for their courses in the fastest and most cost effective way possibie. 

Quaiity is the most important underpinning principie of quaiifications frameworks and 
shouid never be compromised. However, it is important that accreditation processes are 
efficient, fast and not burdened with bureaucratic procedures. 

The potentiai of quaiifications frameworks to underpin the deveiopment of recognition 
of prior iearning (RPL) aiso has not been fuiiy reaiised. Good experience of RPL practice 
has been buiit up in ireiand. However this has been achieved iargeiy on an ad-hoc or 
project basis. More mainstream deveiopment takes time and resources. Poiicy and 
practice shouid focus on areas of demand by expanding on the basis of existing effective 
practices and demonstrated successfui outcomes/benefits reiative to costs. Features of 
an effective 'mainstream' RPL system inciude: 

• One-Stop-Shops for RPL advice, guidance and assessments 

• Dedicated RPL resources, personnei and infrastructure 

• Dedicated RPL 'knowiedge centre' for poiicy, information and promotion. 

At a European ievei, the chaiienges facing quaiity assurance and quaiifications 
frameworks appear more serious. This is not heiped by the confusing number of 
instruments: 

• European Quaiifications Framework (EQF) 

• Quaiifications Framework for European Higher Education Area (Boiogna 
Framework) 

• European Quaiity Assurance in VET (EQARF) 

• European Credit Transfer and Accumuiation System (ECTS) 

• European Credit System for Vocationai Training and Education (ECVET) 

• ESCQ: a new European ciassification of skiiis and competences 

European business weicomes the fact that these toois are expressed in terms of 
knowiedge, skiiis and competence, i.e. iearning outcomes rather than input based 
measures such as duration of studies. However, they aiso needed to be assessed based 
on whether they wiii contribute to improvements in the functioning of iabour markets, 
faciiitate the deveiopment of iifeiong iearning and not iead to unnecessary bureaucracy. 
Measured against these preconditions, it is ciear from the actuai outcomes so far that 
these instruments are stiii far from deiivering. The Copenhagen and Boiogna processes 
cannot continue to deveiop separateiy. 

For member states that are aiready making use of the EQF ieveis, it is necessary 
to evaiuate whether impiementing these has successfuiiy contributed to the use of 
iearning outcomes by education providers, empioyers and empioyees. Sociai partners on 
European and nationai ievei piay an important part in the impiementation process and 
their experience of the tooi so far shouid be assessed. 

There is a high ievei of agreement on the benefits of shifting from a workioad based 
approach in credit and quaiification systems towards one that is based on outcome. 

Such an approach constitutes a step towards greater openness of standards, iearning 
programmes and quaiifications. it foiiows from this added transparency that nationai and 
cross-border staff mobiiity wiii increase as weii. 

For empioyers, buiiding credit and quaiification systems on the actuai outcome from 
iearning wiii make it easier to assess what competencies an individuai possesses. This 
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will in turn contribute to a better match of supply and demand on European labour 
markets. It should be said that employers do not reward qualifications. They reward 
performance. Likewise, the education system should increasingly reward the actual 
outcome from studies rather than for instance the number of weeks a course comprises. 

Progression opportunities from VET towards higher education are important and 
could be facilitated by making more transparent what the learning outcome is of a certain 
VET education. For an individual who wishes to continue into higher education, it should 
be easy to demonstrate to the people responsible for accrediting within the higher 
education system what skills and competencies he or she has acquired. EQF could prove 
a useful tool to increase permeability between VET and higher education credit systems, 
since it makes qualifications more transparent. 

However, it is imperative that enabling such permeability will not lead to an increased 
focus on theoretical elements for VET overall, to the detriment of practical training. This 
is fundamental for VET students to acquire the competencies and skills that will make 
them employable on the labour market. The shift towards an outcome-based learning 
needs to be promoted more strongly also for higher education. This will enable students 
to move more easily between VET and higher education as the qualification and credit 
systems of higher education become more compatible with those of VET. 

It is also important that the European and national qualifications frameworks are 
underpinned by robust quality assurance systems. Trust in the frameworks is essential to 
the guarantee of comparability between awards and of the parity of esteem in the awards 
provided by different bodies. 
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QUALIFICATIONS FRAMEWORKS 
AND LEARNING OUTCOMES 
IN PROGRAMME ORIENTED 
QUALITY ASSURANCE - A BRIEF 
STORY OF DENMARK, AND A FEW 
INTERNATIONAL COMMENTS 

David Metz, ACE Denmark - The Accreditation Institution 


INTRODUCTION 

In 2007 the concept of learning outcomes was introduced into Danish university 
programmes through a ministerial order on a new grading scale (compatible with the 
ECTS scale) to be used in the assessment of students' achievements. It did not enter 
into force until late in 2007, since it required that "the university lays down precise 
descriptions of objectives and criteria for assessing the fulfilment of such objective 
for the individual subjects/subject elements which are concluded with a test"^. The 
requirement of "precise objectives" - another term for intended learning outcomes - for 
each subject was relatively new in a university educational context in Denmark. Up until 
that point educational programmes were typically described in somewhat broader terms 
of objectives^, while subjects were mostly defined by their curriculum. In many ways this 
could be perceived as a small revolution, since students and teachers alike now in theory 
- because of the required description of objectives - were enabled to see the intended 
purpose of every subject in any programme. 

Later in 2007 an act and an order on accreditation and accreditation criteria was 
given, in which it was specified that the learning outcomes of all university programmes 
had to meet the requirements of the new Danish National Qualifications Framework 
for Higher Education (NQF-HE). The NQF-HE was first published in 2008 and finally 
became an official part of the legislation in December 2009 through the latest revision of 
the order on accreditation criteria^ of which the NQF-HE is an appendix. 

RECEPTION OF LEARNING OUTCOMES AND QUALIFICATIONS 
FRAMEWORKS 

There have been no official national studies on the reception of learning outcomes and 
qualification frameworks among stakeholders in higher education in Denmark. A recent 
PhD thesis^^, however, has studied Bologna reforms in Portugal, England and Denmark 
in a comparative perspective. Empirically it is based on interviews with students and 
faculty staff (in physics programmes), as well as document studies and interviews 
with government agency officials and other stakeholders at a national level (carried 
out in 2010). The findings for Denmark here are indifference towards anything Bologna 

9 Ministerial Order no. 250 of 15 March 2007 

10 To be fair, universities had since 2003 worked on implementing learning outcomes on a programme level, using an early 
draft of the Danish national qualification framework, and were since 2004 required by ministerial order to formulate 
"academic and vocational skills" achieved in all programmes. 

11 Ministerial Order no. 1402 of 14 December 2009 

12 Cristina Sin (2012); Loose Policy and Local Adaptation: a Comparative Study of Master Degrees in the Context of the Bologna 
Process, Lancaster University, Department of Educational Research. 
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related amongst faculty staff and students, except for two things: 1) the 3+2 structure of 
university programmes, which is seen as an irrelevant formality since a bachelor degree 
is perceived as having next to no relevance in a Danish context; and 2) the support for 
international student mobility through a structural degree convergence on the European 
level and the ensuing commitment to mutual recognition of qualifications. 

While some teachers in Danish universities find learning outcomes to be a useful tool 
in course planning and student assessment, they reportedly tend to not give them much 
thought while carrying out courses. Most students, on the other hand, seem to be aware 
of the presence of described learning outcomes in the course syllabus as well as on an 
aggregate level in the overall study programme description, but do not perceive them 
as useful in guiding their studying activities - they are more concerned with reading 
prescribed texts and doing prescribed exercises. Contrastingly, interviewees from both 
the Danish Ministry^^ and from Universities Denmark^'^ together with other stakeholders 
at the national level, regard learning outcomes as having had a "huge impact" and having 
been "an enormous administrative task to implement". 

Sin's PhD thesis is a case study, sampling only physics programmes. The findings 
on the perceptions and knowledge of learning outcomes, however, coincide with the 
experiences in the Danish accreditation system, when interviewing students and 
teachers from all types of programmes during site visits. An impression that was 
recently substantiated at a meeting hosted by ACE Denmark in January 2012 with a 
presentation from a representative of the Danish Students' Union, stating that while 
programme learning outcomes could be a relevant add-on guiding students in their 
choice of education, hardly any potential students ever see them today. Furthermore 
a representative from the Confederation of Danish Industry, speaking on behalf of 
employers, said that she sees the programme learning outcomes as mainly a tool to 
be used within the universities and largely useless as a tool for employers. Indeed, 
there seems to be good evidence for the characterisation Sin makes of the Danish 
Bologna implementation as a "top-down" process and effectively resulting in "strategic 
conformity" due to the perceived gains in international student mobility. 

The case could be made for a little less gloomy picture, though. At the aforementioned 
meeting in January 2012 a researcher from a Danish university stressed the potential for 
programme learning outcomes to bridge the gaps between educational programmes, 
students, and employers, while still acknowledging the current pitfalls of e.g. making 
the writing up of learning outcomes a tiresome, bureaucratic exercise of trying to 
describe what is already being done, only now in slightly different terms. Ownership and 
commitment seem to be lacking, so the question now is what can external QA do in this 
respect? 

THE ROLE OF EXTERNAL QUALITY ASSURANCE 

In Denmark there is a very long tradition of external quality assurance of education 
at all levels through the use of external examiners. With the introduction of course 
level learning outcomes and programme accreditation in 2007, a division of labour was 
introduced: 


13 The Ministry of Science, Innovation and Higher Education (current name) 

14 The confederation of all Danish universities 
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EXTERNAL 

EXAMINERS 

EVALUATION TEAMS 
(ACCREDITATION PANELS) 

Measure ALO vs. ILO on 1/3 of all ECTS-credits in 
all programmes 

Programme ILO vs. NQF-descriptors 

Are obliged to report problems with assessment 
practice 

Module ILO vs. Programme ILO 

Report once a year on status, including any given 
problems on ALO levels 

Assessment practices vs. Programme ILO 


External examiners take part in the assessment of students' achievements in minimum 
1/3 of every programme by grading them in collaboration with course teachers. Since 
2007 the grading is done by assessing the achieved learning outcome as compared 
to the intended learning outcome of the subject. Furthermore, the external examiners 
oversee that assessment practice is appropriate and draft up a report once a year on that 
subject. 

The evaluation teams in programme accreditation also evaluate assessment practices 
but as a theoretical analysis by comparing the practices described in a programme with 
the overall learning outcome to make sure there is a minimum of alignment. The validity 
of the analysis is tested during site visits by student interviews. 

The evaluation team further analyses the overall alignment of the (intended) learning 
outcomes at module and programme level, as well as the conformity of the programme 
learning outcomes to the requirements of the NQF-HE at the appropriate level: 


NQF 

Level 

descriptors 


Name of 
degree/award 
- subject(s) 

Illustration of some accreditation criteria and their interconnections 

Some difficulties have been identified thus far: 

1. In some cases the programme learning outcomes are of a very generic nature, 
almost replicating the descriptors of the NQF-HE^® .There is no easy answer to this 
problem. At formal level, the description is aligned with the NQF level, but two 
types of intervention can be possible: in cases of very generic descriptions it can 
be pointed out that the description is not aligned with the name of the programme 
and the award. And since generic descriptions are very broad in scope, it can be 
pointed out that the learning outcomes of the modules do not in full support the 
programme's description. In some cases this can lead to universities revising the 
description of programme learning outcomes. Whether it has any impact on the 
actual planning of the curriculum of the programme remains to be seen. 


Research 

community 


Module ILO 
Admissi. req 
Structure 


Programme 

ILO 


15 The Danish NQF has an absolutely abstract nature because it has to cover all conceivable subjects. 
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2. Evaluation teams can be prejudiced as to what the norms are for programmes 
within a given subject, or more concerned with curriculum and syllabuses than 
with the described learning outcomes of the specific programme. Tell-tale signs are 
usually when team members ask for book lists for a specific course or when they 
openly compare the evaluated programme to programmes they are familiar with 
at other universities. The response to this difficulty has so far been to up-grade the 
training of evaluation team members, among other things focusing on the concept 
of learning outcomes, with the aim of making the teams act as advocates of the 
concept. Increased guidance of team members' compliance with the accreditation 
concept before or after site visits has also been used. 

If the above difficulties do not prevent a relevant analysis, the team tests it in interviews 
with programme management, students, and teachers during a site visit. As indicated 
earlier it is during these interviews that it becomes apparent more often than not, 
that students and often several teachers have only rudimentary knowledge of the 
formulated learning outcomes for subjects and the programme as a whole. Programme 
management, however, are typically more knowledgeable, but they have recently written 
a self evaluation report that should account for the alignment of outcomes. 

The possible effects of programme accreditation, with regards to the implementation 
of learning outcomes and qualification frameworks in Denmark, can in benign cases 
be raising the awareness of the use of learning outcomes and how they can be utilised 
for planning and further developing educational programmes. This is achieved through 
the exercise of writing up the self evaluation report^ and through the interviews with 
the evaluation team. In more problematic cases the effect can be achieved by a more 
direct intervention in the make-up of a programme, by pointing out inconsistencies and 
requiring for them to be amended. 

Whether these stipulated effects are sufficient for stimulating the development of 
commitment and ownership towards learning outcomes (and NQF) for stakeholders in 
university education cannot yet be answered. It seems safe to guess, though, that it will 
probably not be the case for employers. That would require a different approach. 

A FEW INTERNATIONAL COMMENTS 

Most other countries do not have a system similar to the Danish use of external 
examiners, and to many it would not be "culturally acceptable" to try to implement such 
a system. The Danish system relies on some form of measurement of achieved learning 
outcomes. Other countries will have to find other ways of achieving this. The discussion 
in the break-out group touched upon different types of measures: 

1. Career track studies following the career paths of graduates 

2. Survey ofaiumni asking for their self evaluation of learning outcomes 

3. Survey of empioyers asking their evaluation of graduates' proficiency 

4. Pre-graduation survey of students asking for their self evaluation of learning 
outcomes 

5. Measuring the outcome of keystone courses make sample studies of student 
performance 

While all of the above methods may be enlightening, they can only be indirect measures 
of learning outcomes. On the other hand, the Danish model with external examiners is 
expensive and less than bulletproof: it relies on external examiners actually adopting 

16 Assuming the writing of the report is not commissioned from a specialised unit within the university, which sometimes 

seem to be the case. 
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intended learning outcomes as the benchmark for assessing students' achievement, and 
yet they only take part in assessing less than half of the programmes' outcomes. 

So perhaps, if the scale of the potential small revolution is to be investigated seriously, 
the answer lies not in better measurement, but instead in finding new ways of stimulating 
ownership of the concept of learning outcomes. 
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RECOGNITION OF PRIOR 
LEARNING AND THE ROLE OF 
OUALITY ASSURANOE AGENCIES. 
ACCREDITATION OF PRIOR 
LEARNING IN FRANCE AS A CASE 
STUDY 

Teresa Sanchez Chaparro, Programme Manager, Commission des Titres d'Ingenieur (CTI), 
France 


INTRODUCTION 

Recognition of prior learning (RPL) implies the formal acknowledgement of learning 
acquired in a non-formal context (usually coming from experience unrelated to an 
academic context). This process is generally seen as an important tool for progressing 
in the fields of lifelong learning and continuing education, two preeminent objectives of 
the Lisbon agenda. RPL is normally conducted by educational institutions or professional 
certification bodies. In the case of certifications issued by educational institutions, 
this process provides recognition of a certain academic level (according to a national 
qualifications framework) in view of two main objectives: increasing labour market 
recognition and/or enabling access to a higher level of studies. 

Because of its potential role in the fields of employment and social promotion, RPL 
practices are politically sensitive and are normally part of an explicit political agenda 
which responds to national objectives. This political dimension must be taken into 
account by quality assurance agencies as a starting point in order to develop quality 
assurance criteria in this field. The answer to the question: what makes a sound RPL 
process? is fundamentally dependent on the pursued goals and cannot in any way be 
affronted from an exclusively technical perspective. 

An ENQA break-out group session devoted to this subject within the workshop on 
Quality assurance and qualifications frameworks has enabled to confront different 
national realities and policies regarding RPL and, in consequence, different quality 
assurance roles and practices among the different QAA represented. In this paper, the 
French experience in the field of prior learning recognition, or in French terms, validation 
des acquis de I'experlence (VAE), is presented as a case study which enables to illustrate 
some important general issues. After analysing this case, the last section of this paper 
tries to reproduce the main issues raised during the discussion held at ENQA's workshop 
break-out session on recognition of prior learning, and draw some conclusions as to the 
possible role of quality assurance agencies. 
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RECOGNITION OF PRIOR LEARNING IN FRANCE (VALIDATION DES ACQUIS 
DE L'EXPERIENCE- VAE) 

NATIONAL CONTEXT 

RPL processes, as they are practiced today in France, were established by the 2002 Law 
on social modernization^^ with the name of VAE (validation des acquis de I'experience). 

This law establishes an individual right to the recognition of professional experience in 
the acquisition of an academic title or a diploma. This recognition device was established 
within the following context: 

• The existence of a significant population sector with a low or Inexistent graduation 
leveT^. Regardless of their individual competences, this population sector was not 
recognised by the French labour market, and their professional and social ascension 
was, in practice, very difficult. 

• Initial qualifications play a stronger role in France than in other countries (Anglo- 
Saxon countries, for example) in reference to social hierarchy. Some authors talk 
about a 'French structural fondness for qualifications'^^ or even about a 'French 
dictatorship of qualifications'^” . A 2005 study shows that over 73% of job offers 
in France, regardless of the age of the candidate, mention the number of years 
of study and the qualification required (in Great Britain, this rate goes down to 
27%)^'. Different studies also show that the possession of a diploma enables a 
quicker and more stable access to the job market in France. The fact of having a 
diploma has also an impact on the stability of the contract (with a much higher rate 
of indefinite contracts for graduates)^^. Finally, the possession of an initial academic 
qualification plays, at least since Napoleonic times, an important symbolic role in 
the meritocratic France, where initial qualifications are one of the axes for social 
differentiation. 

• A dual training system which establishes a sharp distinction between initial training and 
continuous and vocational education training (CVET). As an example, only around 

1,5 % of continuous education actions in 1996 resulted in an academic title or 
diploma^T 

The new VAE system established by the 2002 law tries to profit from the major role 
played by initial diplomas in France in order to attain two main objectives: 

• Facilitate social promotion and secure career paths 

• Build a bridge between the CVET system and the initial training system. 

BRIEF DESCRIPTION OF THE FRENCH VAE SYSTEM 

The VAE process enables an individual to get all or part of a certification (diploma 
or professional qualification certificate) based on his/her professional experience 
(salaried, non-salaried, or voluntary). This experience, which must be related to the 
intended certification, is validated by a panel. If the VAE process leads to an academic 


17 Law n° 2002-73 du 77 janvier 2002 de modernisation sociale 

18 According to the 1995 INSEE (Institut National de la Statistique et des Etudes Economiques- National Institute for 
Statistics and Economic Studies) employment survey, four out of ten workers in France didn’t have any formal qualification 
at the time. 

19 TRY BY, Emmanuel; Accreditation of prior experiential learning and the development of higher education; European journal 
of vocational training, 46- 2009/1-IS5N 1977-0219 

20 Mehaut Philippe, Lecourt Anne-Juliette (2007) ; The accreditation of prior learning experience in France : A good start, to 
be continued ; Discussion paper. 

21 Marchal Emmanuelle, Rieucau Geraldine (2003) « Candidat de plus de 40 ans non diplome ou debutant s'abstenir », 
Connaissance de I'emploi n°11, CEE 

22 Studies made by the CEREQ (Centre d'Etudes et des Recherches sur les Qualifications - Center for the study and research 
on qualifications) 

23 Again from the studies by the CEREQ 
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qualification, it is directly assessed and recognised by the higher education institution. 
The diploma obtained has exactly the same validity as a diploma obtained by an 
ordinary study path. All diplomas, certificates, and professional qualifications are 
available through VAE. They are registered in a public registry (RNCP- National Register 
of Professional Certifications, http://www.rncp.cncp.gouv.fr/). Academic diplomas 
registered in RNCP cover a variety of levels, from secondary education to the masters 
level. All engineering degrees (masters level diplomas) are included by law in the RNCP 
and have thus the obligation of delivering their diplomas also through VAE. 

The diagram below shows the main steps of the VAE process and the main actors 
involved. Several structures at the regional and national level have been put in place 
in order to provide general information on the procedure and to orientate candidates 
towards a suitable certification. Once the certification is chosen, a first check on the 
admissibility of the request (mainly the verification of the candidate having the three 
years of relevant experience required) is conducted. After passing the admissibility 
check, the candidate must prepare an application which will be finally examined by a 
mixed panel composed of academic and professional members. The panel may issue 
a full or a partial validation. In the latter case, the jury can propose the necessary 
pedagogical complements (courses, internships, and so on). There is no time limit for 
obtaining these additional competences. 


Counselling structures 
(National and Regional level) 


General information on VAE 
Orientation towards a suitable certification 


HEI 


Admissibility of demand (3 years 
of relevant experience) 


HEI or another agreed/labellised 
organism 


Guidance during the process 
Help for preparing the application 


HEI 


Study of demand by the jury 


No validation Full validation Partial validation 

THE VAE PROCESS IN ENGINEERING HIGHER EDUCATION IN FRANCE: THE 
ROLEOFCTI 

The main mission of CTI is to conduct accreditation of engineering programmes in 
France. CTI supervises all paths for obtaining an engineering diploma, including VAE. All 
engineering institutions must include a specific section concerning VAE in their self- 
assessment report in view of programme accreditation (VAE procedures, number of 
accreditations issued, number of candidates, cost of the process, type of compensatory 
measures proposed, etc.). 

CTI has developed a set of criteria regarding the way VAE procedures should be 
conducted. These criteria are gathered in the document References et Orientations 
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(References and Guidelines- available at www.cti-commission.fr). The quality criteria 
applied are coherent with the political goals established at the national level. In 
consequence, the criteria have been developed under the assumption that VAE is 
basically a beneficial procedure, with a strong potential to boost employment and social 
advancement. Engineering institutions in France are sometimes accused of being elitists, 
and VAE is considered by CTI as one of the devices which can be employed to foster 
social diversity in this environment. 

The legal obligation for all HEIs to establish VAE processes has also had some 
methodological benefits at the national level, in the sense that it has been a major driver 
to formalise the competence-based approach within French higher education institutions. 
Some engineering institutions in France are reluctant to deliver their diplomas through 
VAE, as they think that the educational value of the VAE is lower than a classical diploma 
and hence could degrade the value of their academic certificates. Being able to articulate 
a VAE procedure that ensures the same level of exigency as in the standard path is 
indeed a crucial issue for CTI. 

At the other end of the spectrum, some higher education institutions could take this 
practice as a business opportunity. The fact that they can deliver a "partial validation" 
followed by a recommendation on pedagogical complementary elements could 
encourage engineering institutions to use VAE in order to fill their continuous education 
programmes. CTI is aware of this potential conflict of interest and tries to be vigilant in 
order to prevent abuse. 

As far as accreditation is concerned, CTI has tried to adopt a balanced position, which 
respects the specific policies of each institution with regards to VAE but, at the same 
time, tries to ensure a fair implementation of these recognition processes. The main 
accreditation criteria applied by CTI are: 

• Certification: the diploma delivered must be strictly the same as that delivered 
through other learning paths. 

• Professional outcomes of the programme: engineering institutions must adequately 
formalise the outcomes of their programmes in a way that they can serve the 
certification of professional competences. These descriptions must be made public 
and transmitted to the National Registry of Professional Certifications. 

• Evaluation procedures and criteria: the procedures and evaluation criteria must be 
clear and public. They must guarantee an equitable treatment of all candidates. 

The evaluation criteria should be at the same exigency level than the ones used to 
attribute the diploma through other paths. 

• Information and guidance to the candidate: institutions must be transparent and 
provide sufficient information to possible VAE candidates. They must put in place 
(or offer) guidance and counselling to the candidates throughout the process. 

• Accreditation panel: the VAE panel must include academic and professional 
members (other than the external lecturers of the programme). It must include 
some members from the jury that delivers the ordinary diploma in order to ensure 
a same exigency level, but other than that, the composition must be substantially 
different and adapted to understand the specific challenges of this track. 

CONCLUSIONS 

Recognition of prior learning is implemented at various levels and with various objectives 
in the different European countries. Whereas the RPL process in some countries^'* mainly 
involves considering the learner's prior formal or informal learning in order to gain entry 


24 Such as Ireland, Germany or the Netherlands 
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to further study or to gain credit or exemption towards a degree, in other countries, such 
as France, RPL is used as a major route to award an academic degree. 

The case of France's VAE is indeed an extreme case. VAE enables the acquisition of a 
complete academic degree through the exclusive validation of professional experience. 
The VAE legislation does not limit the number of certifications obtained by a particular 
person. In France, it is hence possible to get up to the master level without having any 
formal academic qualification. As an example, we can examine the case of a French 
citizen, who left school without his secondary education degree and was subsequently 
able to obtain 6 academic diplomas through VAE (including a bachelor in literary 
Arabic) up to the master level. This person is currently in charge of VAE processes at a 
prestigious engineering institution^. 

The French approach to RPL raised some critical voices among QA colleagues 
during the ENQA workshop break-out group discussion. Whereas the break-out group 
participants could easily accept that one can arrive to the same level of professional 
competence through working experience as via an academic degree, it is more difficult 
to argue that this path could be equivalent in terms of methodological skills and analytic 
capacity. There is also an experiential dimension associated to formal higher education 
which definitely plays a role in building a competence profile. 

The case of Hong Kong^® provides an example of how RPL processes can be 
established without mixing the notions of level and profile. In Hong Kong the recognition 
of professional competences is made by a number of specific agencies with close links 
to industry. Candidates obtain recognition of a certain academic level and the right to be 
admitted to a higher level of studies, but they do not obtain an academic degree. 

However, the French VAE should be understood in the light of the French specific 
national context, namely a number of urgent structural problems which needed to be 
addressed in order to improve competitivity and social equity, and the special role played 
by initial qualifications. 

RPL practices are indeed politically sensitive and highly dependent on the context. 
Agencies must explicitly assume this political dimension in order to define what role to 
play regarding RPL processes. The following questions should be posed: 

• Is RPL an important issue in my specific national and political context? Is it being 
practiced at a significant level? 

• Are there any risks of derive in the way HEIs are implementing these recognition 
processes? 

Depending of the answer to these questions, the quality assurance agency may adopt 
a more or less active role. In the case of France, RPL practices are politically important 
and there are indeed certain risks - such as a general resistance of institutions to VAE; 
conflicts of interest to be avoided; and certain methodological aspects associated to 
the competences approaches - which justify, as we have seen, an active role from the 
national accreditation agency. 


25 See for example, the case of Hassane Akka; 
http://orientactuel.centre-inffo.fr/Hassane- Akka-de-bac-3-a-bac-5-en.html 

26 http;//www.h kqf.gov. hk/guie/HKQF_intro. asp 
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ASSESSMENT OF LEARNING 
OUTCOMES: THE CASE OF ANECA 

Rafael Llavori, Head of Unit for International Relations, ANECA, Spain 

INTRODUCTION 

The assessment of learning outcomes (LO) has become a major element in the last 
five years in the quality assurance and accreditation approaches throughout the 
European Higher Education Area. They play a major role in the evaluation of the study 
programmes. The challenge is the definition of clear and useful tools, which allow 
agencies to identify that the programme has in place mechanisms that assure the correct 
achievement of the different types of learning outcomes defined in the programme. To 
be sure that planning journey that comes from the intended learning outcomes defined 
in the programme proposal, to the achieved learning outcomes attained by the students 
according that proposal has been properly chartered. 

We could say that the approach must be necessary tri-dimensional. It means that 
we need to bear in mind the 'length' defined by the scope of the evaluation; a 'width' 
represented by the national/regional/specific context; and the 'height', represented by 
the nature of the institutions involved: programme or institutional. There is a general 
agreement on the fact that it is not possible to identify successful practices in a particular 
context and trying to transpose them into a different context. The contexts of the 
higher education system certainly matters. Here is where the different approaches 
accomplished by the agencies at the European level can provide us with a useful range 
of good practices to be collected and the different experiences of the higher education 
institutions in implementing these processes are definitely meaningful. 

OUTCOMES OF THE BREAK-OUT GROUP ON I - ASSESSMENT OF 
LEARNING OUTCOMES 

The main focus of the breakout group was to exchange the current experiences 
developed in the different settings represented at the table. As an introduction for the 
further debate the Spanish example represented by the National Agency for Quality 
Assurance and Accreditation, ANECA, was explained, underlining the challenges for both 
agencies and institutions 'in the making'. 

THE ACCREDITATION PROCESS IN SPAIN: THE ASSESSMENT OF LEARNING 
OUTCOMES 

In Spain the legal framework establishes programmes accreditation in a three-stage 
process comprising ex-ante accreditation, a follow-up procedure throughout the 
implementation of the programme and an ex-post accreditation in six years for bachelors 
and four years for masters. 
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THE ACCREDITATION PROCESS IN SPAIN 

The Spanish legal framework establishes three different stages in the accreditation process 


Stage 1 EX-ANTE ACCREDITATION (Verificacion) 


Stage 2 FOLLOW-UP PROCEDURE (Seguimiento) 


stages EX-POST ACCREDITATION (Accreditacion) 


1. Ex-ante accreditation (verificacion in Spanish): to verify that the design of the 
programme includes the criteria established by the Ministry. A public report is 
delivered by ANECA. 

2. Follow-up procedure (seguimiento): to check, by non-intrusive means, that the 
institution is implementing the programme according to the approved design based 
on a certain number of indicators. 

3. Ex-post accreditation (accreditacion): to certify that the programme has been 
accomplished according to the already approved conditions in the ex-ante 
accreditation process. 

The Spanish Qualifications Framework (MECES) was established in 2010. The legal 
framework integrates it as part of the evaluation process regarding competences. 
Furthermore, the Dublin Descriptors are part of the legal framework as the reference 
points for the bachelor, master and doctoral levels. Designs for new degrees have to 
be evaluated according to a series of quality-based criteria. The ex-ante accreditation 
procedure defines ten criteria: 

1. Description of the degree 

2. Justification 

3. Competences 

4. Student entry and admission 

5. Programme planning 

6. Academic staff 

7. Resources and services 

8. Anticipated outcomes 

9. Quality assurance system 

10. Agenda for implementing the degree 

Criteria number 3, 5 and 8 are the relevant ones in terms of intended learning outcomes 
and are the key ones to be followed in the ex-post accreditation procedure. 

The real challenge for the Spanish approach to the assessment of learning outcomes 
is twofold. For the QA agencies, matching the achieved learning outcomes to the list of 
intended ones and to develop effective and coherent tools to assess achieved learning 
outcomes leaving enough room to the programmes to avoid the definition of a ready-to- 
use recipe or a sort of, in this very case 'ANECA Companion to the assessment of learning 
outcomes'. 
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For the higher education institutions, the main challenge has been the need to work 
out their agendas for adapting their teaching-learning experiences on the learning 
outcomes- approach from intended to achieved learning outcomes, and developing 
feasible assessment mechanisms to assure their attainment by the students. For the 
QA agency the challenge is to be able to develop tools to orient the learning outcomes 
assessment towards the work done by the institutions. The goal of the agency is to 
guarantee that the programme as a whole reached its objectives defined in the ex-ante 
accreditation in terms of achieved learning outcomes. 

CONCLUSIONS 

The main conclusions of the lively debate that took place in the break out group can be 
summarised as follows: 

• Need to analyse the assessment of learning outcomes paying attention to the legal 
framework and academic context 

• The maturity of the higher education system as a whole, and the implementation of 
other 'Bologna tools', such as qualifications frameworks, plays a significant role in 
the process 

• The maturity of the higher education institutions in implementing ways of assessing 
learning outcomes 

• The maturity of QA agencies in developing mechanisms to evaluate learning 
outcomes in a transparent, reliable and coherent way 

• Need to focus on the quality assurance of the assessment procedures of learning 
outcomes: check the assessing practices of the programmes to assess different 
learning outcomes (consistency) 

• Need to incorporate opinions from external academic staff and questioning 
whether it is possible to incorporate 'external views' from graduate students and 
employers, too 

• The focus is on the programme learning outcomes, not on the individual student 

As a final conclusion it has to be stated that the most positive outcome of the session 
was likely the possibility to discuss and deal with experiences which are currently being 
implemented and being tested and to be able to do it through drawing on the daily 
experience and challenges from both agencies and higher education institutions. 
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CONCLUSIONS 

Several current reforms in higher education are having a significant impact on quality 
assurance and the work of quality assurance agencies. Among these, the establishment 
and referencing to qualifications frameworks and the adoption of assessment methods 
focusing on student learning outcomes pose important challenges to the methods and 
processes used for internal and external quality assurance. 

The ENQA 2012 workshop on Quality Assurance and Qualifications Frameworks: 
exchanging good practice and these articles covered a wide range of issues related to 
qualifications frameworks, such as the state of art in their development and adoption, 
the role of agencies in the self-certification process, and the impact of qualifications 
frameworks on external quality assurance. In addition, a view from the stakeholder 
community was presented through an article outlining the use and importance of 
qualifications frameworks to the labour market and employers. Recognition of prior 
learning, assessment of learning outcomes, and learning outcomes in programme 
oriented quality assurance were also discussed in smaller working groups during the 
workshop. 

As the first article underlines, it is clear that while several changes have already 
taken place, we are still at the beginning of implementing qualifications frameworks. 

It is important to make sure that they are developed jointly with quality assurance, 
learning outcomes, and other recognition tools. At the same time, the article emphasises 
the importance of making sure that European, but especially national qualifications 
frameworks become more visible and better known, so as to bring real benefits to 
the end users, whether individuals, higher education institutions, academic staff, or 
employers. 

In terms of self-certification, the second author underlines that verification by self- 
certification is a process in which each QA agency has a significant and obligatory role 
to play. The role of quality assurance is to demonstrate that programmes are based on 
intended learning outcomes and that qualifications are awarded on basis of achievement 
of these outcomes. If, however, we are to realise the objectives of the QF-EHEA in 
relation to transparency, mobility, and recognition, self-certification is only the first step. 

The Swedish case example presents a recently adopted approach to quality assurance. 
In line with the Bologna Process and the goal of increased employability of students, 
qualification descriptors were introduced and have become the tool in quality assurance 
of higher education in Sweden. Continuous reviewing in the coming years will help to 
develop best practice in implementing the new approach, and will be able to provide 
informed answers to questions such as: Can results of an academic study programme be 
measured? If yes: how can that be done? And is this European quality assurance of the 
21st century? 

The system used in Denmark relies on the use of external examiners for the 
measurement of achieved learning outcomes. Discussions on the use of learning 
outcomes in programme based QA in the related working group brought up a number of 
different methods. A conclusion of the group's work was that if the scale of the potential 
small revolution brought about by learning outcomes based assessment is to be 
investigated seriously, the answer lies perhaps not in better measurement, but instead in 
finding new ways of stimulating ownership of the concept of learning outcomes itself. 

The working group on recognition of prior learning discussed different ways in which 
prior learning is used for entry into further study, achievement of credits, or for the 
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award of an entire degree. It became clear that recognition of prior learning practices is 
politically sensitive and highly dependent on the national context. It was felt thus that 
agencies must explicitly assume this political dimension in order to define what role to 
play regarding recognition of prior learning processes in their own frameworks. 

The working group discussing the assessment of learning outcomes concluded that 
there is a need to analyse the assessment of learning outcomes paying attention to the 
legal framework and academic context. In addition, the maturity of the higher education 
system as a whole, and the degree of implementation of other 'Bologna tools', such as 
qualifications frameworks, play a significant role in the process. The group felt a need 
to focus on the quality assurance of the assessment procedures of learning outcomes 
through checking the assessment practices used by programmes to assess different 
learning outcomes. The participants agreed that the focus of such assessment should be 
on the programme learning outcomes, not on the achievement of individual student. 

Overall, the main conclusion of the workshop and of this publication is that there 
is great benefit in sharing and comparing national practices, and learning from good 
practice at the level of quality assurance agencies. However, the national political and 
legal context, as well as the degree of implementation of the Bologna reforms, has a 
significant impact on the way in which agencies can and should react and relate to the 
implementation of qualifications frameworks. Coordinating efforts in developing and 
implementing qualifications frameworks and other Bologna reforms is important to 
ensure a successful consolidation of the European Higher Education Area. In addition, all 
relevant actors should make efforts to ensure that information on the role and purpose 
of qualifications frameworks and learning outcomes-based assessment is provided to all 
interested parties, including employers, so that student employability can be improved, 
mobility facilitated, and recognition of non-formal and informal learning further 
developed. 
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ANNEX I 

ENQA WORKSHOP - QUALITY ASSURANCE AND QUALIFICATIONS 
FRAMEWORKS; EXCHANGING GOOD PRACTICE 

CO-HOSTED BY FETAC, HETAC, lUOB AND NOAI 
9-10 FEBRUARY 2012 

Venue 

The Bedford Hall, Dublin Castle 
Dame Street 
Dublin 2, Ireland 

PROGRAMME 
Thursday 9 February 

12.00 Registration of participants/lunch 

13.00 Welcome, Acti;m Hopbach, President of ENQA 

Session I Chair: Achim Hopbach 

13.10 Introductory plenary session 

Qualifications frameworks in the EHEA: the state of development and 
perspectives for the future 

Carita Blomqvist, Finnish National Board of Education 
Discussion 

13.50 Self certification of Qualifications Frameworks: The role of QA agencies 
Bryan Maguire, HETAC 

Discussion 

14.35 Role and meaning of Qualifications Frameworks in external quality 

assurance 

Karin Jarplid Linde, National Agency for Higher Education, Sweden 
Discussion 

15.20 Coffee break 

15.50 Breakout groups 

Group I: Assessment of learning outcomes - exchanging national 

practice and experiences - Chair: Rafael Llavorl, ANECA 

Group II: Recognition of prior learning- exchanging national practice 

and experiences - Chair: Teresa Sanchez, CTI 

Group III: Working with learning outcomes and NQFs in quality 

assurance: A practical interactive session- Karena Maguire, HETAC and 

Mary Sheridan, HETAC 

17.30 End of Day 1 

Qptional visit of Dublin Castle (30 min) 

19.00 Workshop Dinner 
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Friday 10 February 

Session II Chair: Marla Kelo, Director, ENQA 

9.00 Stakeholder view on the use of qualifications frameworks and learning 
outcomes in external QA 

Tony Donohoe, the Irish Business and Employers Confederation 
Discussion 

09.45 Coffee break 

10.15 Breakout groups 

Group I: Qualifications frameworks and learning outcomes in 

programme oriented QA - Chair: David Metz, ACE Denmark 

Group II: Qualifications frameworks and learning outcomes in 

institution oriented QA - Chair: Sarah Butler, QAA 

Group III: Working with learning outcomes and NQFs in quality 

assurance: A practical interactive session- Karena Maguire, HETAC and 

Mary Sheridan, HETAC 

Session III 

12.00 Qualifications frameworks and learning outcomes in practice: 

Good practice and challenges 

Chair: Padraig Walsh, CEO, FETAC, HETAC, lUQB, NQAI 

13.00 End of Workshop/Lunch 
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ABOUT ENQA 

The European Network for Quality Assurance in Higher Education was established in 
2000 to promote European co-operation in the field of quality assurance. In November 
2004 the General Assembly transformed the Network into the European Association for 
Quality Assurance in Higher Education (ENQA). 

The mission of ENQA is to contribute significantly to the maintenance and 
enhancement of the quality of European higher education at a high level, and to act as 
a major driving force for the development of quality assurance across all the Bologna 
signatory countries. 

ENQA's purposes are essentially threefold: 

• to represent its members at the European level and internationally, especially 
in political decision making processes and in co-operations with stakeholder 
organisations; 

• to function as a think tank for developing further quality assurance processes and 
systems in the EHEA; 

• to function as a communication platform for sharing and disseminating information 
and expertise in quality assurance among members and towards stakeholders. 


ENQA Secretariat 

Avenue de Tervuren, 38 - bte 4 
1040 Brussels, Belgium 
Tel:-r32 2 735 56 59 
Fax: +32 2 735 61 53 
E-mail: secretariat(5)enqa.eu 
Website: www.enqa.eu 
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THE THEME of qualifications frameworks and their relation to quality assurance is gaining urgency in the European scene 
as more and more countries are completing their national qualifications frameworks and quality assurance agencies need 
to take important decisions on how to implement them. This report is based on the ENQA workshop "Quality Assurance 
and Qualifications Frameworks: exchanging good practice”, which took place in Dublin, Ireland, on 9-10 February 2012 
and presents articles on themes such as the state of development of qualifications frameworks, the role of agencies in the 
self-certification process, and the meaning of qualifications frameworks in external quality assurance.. 


ENQA 

WORKSHOP REPORT 21 

ISBN 978-952-5539-69-1 (paperbound) 
ISBN 978-952-5539-70-7 (pdf) 
ISSN1458-106X 


